Papers by Drew A. Hudson

Copied to clipboard

Challenge:	Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them.
Approach:	They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering.
Outcome:	The proposed model improves the original BERT model on downstream tasks by large margins.